
The Patent Office 
Concept House 



South Wales 
NPIO 8QQ 



Cardiff Road 
Newport 



INVESTOR IN PEOPLE 



I, the undersigned, being an officer duly authorised in accordance with Section 74(1) and (4) 
of the Deregulation & Contracting Out Act 1994, to sign and issue certificates on behalf of the 
Comptroller-General, hereby certify that annexed hereto is a true copy of the documents as 
originally filed in connection with the patent application identified therein. 



In accordance with the Patents (Companies Re-registration) Rules 1982, if a company named 
in this certificate and any accompanying documents has re-registered under the Companies Act 
1980 with the same name as that with which it was registered immediately before re- 
registration save for the substitution as, or inclusion as, the last part of the name of the words 
"public limited company" or their equivalents in Welsh, references to the name of the company 
in this certificate and any accompanying documents shall be treated as references to the name 
with which it is so re-registered. 



In accordance with the rules, the words "public limited company" may be replaced by p. I.e., 
pic, P.L.C. or PLC. 



Re-registration under the Companies Act does not constitute a new legal entity but merely 
subjects the company to certain additional company law rules. 



Signed WV {^^Ac,^ 

Dated ^ dautnjjtf O^OO 




An Executive Agency of the Department of Trade and Industry 



THIS PAGE BUNK (USPTO) 



mum E55829^-3 D02611 
POl/7700 0.00-0019082.7 




The Patent Office 

Cardiff Road _ 
T' . Newport^ * 
* / GwentNP9lRH ; - ^ , 



1. 



Your reference 



2636202/DS 



2. 



Patent Application Number 



0019082:7 



-5 AUG 2000 



Full name, address and postcode of the or of each applicant (underline all surnames) 
Canon Kabushiki Kaisha 

30-2 3-Chome Shimomaruko _ ^ 

Ohta-Ku 

Tokyo 

Japan 



Patents ADP number (if known) \o O O S 

If the applicant is a corporate body, give the Country: JAPAN 

coimtry /state of its incorporation State: 



4. Title of the invention 



METHOD AND APPARATUS FOR GENERATING MODEL DATA FROM CAMERA 
IMAGES 



5. 



Name of agent 



Beresford & Co 



"Address for Service" in the United Kingdom 
to which all correspondence should be sent 



2/5 Warwick Court 
High Holborn 
London WCIR 5DJ 



Patents ADP number 



6. 



Priority details 



Coimtry 



Priority application number 



Date of filing 



Patents Form 1/77 



7. 



If this application is divided or othenvise^derived from an earlier UK application g'iye details 
Number of earlier application : ; f^^^ n v ' v V , •* ' 



requeltr^^^^^ and or right to grant of a patent reqimed in supp this 



YES 



9. Enter the nmnber of sheets for any of the foUowing items you are filing with this form." 

Continuation sheets of this form 0 

Description 65 , , , 

Claim(s) 27 \ 

Abstract ' 2 ' * 

Drawing(s) 23 * ^ 



10. If you are also filing any of the following, state how many against each item. 

Priority documents 0 
Translations of priority docmnents 0 
Statement of inventorship and 

right to grant of a patent (Patents form 7/77) 1 + 3 copies 

Request for preliminary examination 

and search (Patents Form 9/77) Q 

Request for Substantive Examination 

(Patents Form 10/77) 0 

Any other documents 

(please specify) q 



11. lAVe request the grant of a patent on thebasis of this application 





12^ Name and daytime telephone number of David SPROSTON 

person to contact in the United Kingdom 

Tel: 020 7831 2290 



Patents Forr^ 7^ 
Patents Act i;9 

(Rule 15) 



Statement 
right to grant 




The Patent Office 

Cardiff Road 
Newport 

Gwent NP9 1RH 



1. 



Your reference 



2636202/DS 



2. Patent Application Number 

accompanying application reference 2636202 



3. Full name of the or each applicant 

Canon Kabushiki Kaisha 



0019082.7 



5 AUG 2000 



Title of the invention 

METHOD AND APPARATUS FOR GENERATING MODEL DATA FROM CAMERA IMAGES 



State how the appiicant(s) derived the right from the inventor(s) to be granted a patent 

By virtue of the employment of the inventors by Canon Research Europe Ltd, and by virtue of 
an agreement betvi^een Canon Research Europe Ltd and Canon Kabushiki Kaisha dated 
1 January 1994. 



6- How many, if any additional Patents Forms 

7/77 are attached to this form? 

None 



lAVe believe that the person (s) named over the page (and on any extra copies of this form) is/are the 
inventor(s) of tbe i nvention which the above,pa^nt application relates to. 



Signature 





^r^ERESFG^RD & Co 



Date 3 August 2000 



8. 



Name and daytime telephone number of 
person to contact in the United Kingdom 



David SPROSTON 
Tel: 020 7831 2290 



Patents Form 7/77 



HAS LAM; Jane 




7 Moreton Place 




Twynersh Avenue 




Chertsey 




Surrey 




KT16 9DE 




UNITED KINGDOM 









TAYLOR; Richard Ian 

c/o CANON RESEARCH CENTRE EUROPE LTD 

1 Occam Courts Occam Road 

Surrey Research Park 

Guildford 

Surrey 

GU2 5YJ 

UNITED KINGDOM 

^^"gy^—f oT 



WILES } Charles Stephen 

5 Sandalwood 

Guildford 

Surrey 

GU2 5NZ 

UNITED KINGDOM 



1 

METHOD AND APPARATUS FOR GENERATING 
MODEL DATA FROM CAMERA IMAGES 
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The present invention relates to an apparatus and method 
5 of operation thereof for generating model data of a model 
in a three-dimensional space from image data 
representative of a set of camera images of an object. 

It is known to create three-dimensional computer models 
10 of real objects based on the input of image data in the 
form of a series of image frames which may be derived 
from a series of photographs taken from different camera 
positions or from a video recording taken from a moving 
camera . 

15 

Having generated a set of model data, a model image is 
displayed and may be compared with camera images of the 
object from which the existing model data has been 
derived. 

20 

A first aspect of the present invention relates to 
refinement of the existing model data by allowing a user 
to identify an additional feature in one of the camera 
images, this feature being absent from the displayed 
25 model image, and which the user wishes to include in the 
model by the input of additional model data. 
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One method of refining the model in this respect requires 
the user to continue the process of entering matching 
points identified in successive image frames and- the 
apparatus to then process the matching point- data by re- 
running the model program to incorporate an expanded set 
of data- This process however requires a substantial 
amount of computer processing effort with consequent 
delay. In some instances, the additional feature may 
only be visible in a single frame, making it impossible 
to identify a matching point in a second frame. 

The present invention seeks to provide an dlmproved method 
and apparatus allowing an additional feature to be added 
as a result of user input based on a single frame. 

According to the present invention there is disclosed a 
method of operating an apparatus for generating model 
data representative of a model in a three dimensional 
space from image data representative of a set of camera 
images of an object; 

the apparatus performing the steps of; 

displaying a model image based on an existing set of 
model data; 

displaying one of the ceunera images of the object 
for selection by a user of an additional feature to be 
represented by additional model data; 

receiving an image point selection signal responsive 



2. 

3 263620^ 
to user actuation of an input means and identifying co- 
ordinates of an image point in the camera image defining 
the selected additional feature; 

calculating a locus in the three dimensional space 
defining positions of possible model points corresponding 
to the image point and consistent with the geometric 
relationship between the object and a camera position 
from which the displayed camera image was taken; 

displaying a position indicator in the model image 
at co-ordinates in the model image corresponding to one 
of the possible model points on the locus; 

receiving positioning signals responsive to user 
actuation of the input means and updating the co- 
ordinates of the position indicator such that movement of 
the position indicator is constrained to follow a 
trajectory in the model image corresponding to the locus; 

receiving a model point selecting signal responsive 
to user actuation of the input means and determining 
selected co-ordinates of the position indicator to be the 
position indicator co-ordinates at the time of receiving 
the model point selecting signal; and 

determining co-ordinates of the additional model 
point in the three dimensional space corresponding to the 
selected co-ordinates of the position indicator. 



In a preferred embodiment, the locus is a straight line 
in the three-dimensional model space, the straight line 



4 263620\ 

being displayed in the model image as a visual aid^to the 
user., in : editing the position of . the ..new niodel point- 
After! finalising the position of ^the new/^mddeL- pointer ^a .^ 
model generating process is initiated to incorporate the 
additional model point into the model data and to 
generate surface elements of the model, allowing the new 
model image to be displayed including the surface 
elements for comparison with the camera image. 

A second aspect of the present invention relates to the 
manner in which the model data is edited when a new model 
point is added to the existing set of model data, either 
using the above disclosed method or by other methods. 
Incorporation of an additional model point generally 
requires the surface elements of the existing model to be 
modified, at least one of the surface elements being 
replaced by a plurality of new elements which include the 
new model point. This. aspect of the invention addresses 
the problem of selecting the surface element to be 
modified or replaced in a manner which is simple for the 
user to implement. 

According to the second aspect of the present invention 
there is disclosed a method of operating an apparatus for 
generating model data defining a model in a three 
dimensional space, the model data comprising co-ordinates 
defining model points and surface elements generated with 
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reference to the model points; the method comprising 
editing an existing set of model data by the steps of ; 

adding a new model point to the existing set of 
model data; 

projecting the new model point onto the model and 
identifying a selected one of the surface elements onto 
which the new model point is projected; 

identifying a subset of the model points which 
define the generation of the selected surface element; 

adding the new model point to the subset to form an 
edited subset of model points; and 

generating one or more edited surface elements from 
the edited subset of model points to replace the selected 
surface element* 

The identification of the surface element to be replaced 
is thereby automatically implemented by the apparatus^ by 
operating a computer program selected by the user. 

In a preferred embodiment, the projection of the new 
model point onto the model is processed by defining a 
centre of projection corresponding to one of the camera 
positions from which frames of the ccunera image data were 
obtained- An interface allowing the user to select an 
appropriate camera position may comprise a display of a 
pictorial representation showing the relative positions 
of the object and the cameras, the camera positions being 
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represented by icons which may be selected by clicking a 
computer mouse or other input device. 

A further embodiment provides an alternative interface in 
which thumbnail images of the camera image frames are 
presented to the user, each thumbnail image constituting 
an icon allowing selection using a pointing device such 
as a computer mouse in conjunction with a moveable cursor 
on the display screen. 

A third aspect of the preisent invention relates to the 
need to enable the user to evaluate the quality of a 
model in order to judge whether further refinement of the 
model data is required and to judge whether any editing 
procedure has been correctly effected or requires further 
editing - 

This aspect of the invention seeks to provide the user 
with an interface allowing the user to view a model image 
for comparison with a caimera image, it being advantageous 
to present the user with compatible views for ease of 
comparison. The selection of the appropriate model image 
for comparison with a specific camera image may be time 
consuming and complex to the user- 
According to the present invention there is disclosed a 
method of operating an apparatus for generating model 
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data representative of a three dimensional model, of an 
object from input signals representative of a set of 
camera images of the object taken from a plurality of 
camera positions, the method comprising; 

displaying a set of icons, each being associated 
with a respective one of the camera images of the object; 

receiving a selection signal responsive to user 
actuation of an input means whereby the selection signal 
identifies a selected one of the icons; 

determining a selected camera image from the set of 
camera images corresponding to the selected icon; 
displaying the selected image; 

determining position data representative of a 
selected camera position from which the selected image 
was taken; 

generating in accordance with said model a model 
image representative of a view of the model from a 
viewpoint corresponding to the position data; and 

displaying the model image for visual comparison' 
with the selected image by the user. 

This method therefore allows the user to simply select a 
camera image using a set of icons and provides automatic 
processing using a computer program to generate a model 
image representative of a view of the model from a 
viewpoint corresponding to position data determined when 
the user selects a particular icon.. 
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The icons may . be representations of camera portions 
relative to a representation of the object being modelled 
or alternatively the icons may be thumbnail, images of .the: 
frames of ccunaera image data. 

The user is thereby presented with a computer interface 
allowing correctly comparable model and camera images to 
be rapidly selected for evaluation. The selection 
process may thereby be repeated to view the images from 
different viewpoints in order to rapidly gain an overview 
of the quality of the model data as a basis for deciding 
whether further editing is required. 

Embodiments of the above aspects of the present invention 
will now be described by example only and with reference 
to the accompanying drawings of which; 

Figure 1 is a schematic representation of components of 
a modular system in which the present invention may be 
embodied; 

Figure 2 is a schematic representation of a model window; 

Figure 3 is a schematic representation of a ccimera image 
window in which a displayed camera image includes an 
additional feature which is not represented in the model 
image of Figure 2 ; 



2- 

9 2636201s 
Figure 4 is a schematic representation of a calculated 
locus in the 3-D model space for a new model points- 
Figure 5 is a schematic representation of a model window 
including a new point moved by the user to positions 
constrained by the calculated locus; 

Figure 6 is a schematic representation of a model window 
during user selection of points for connection to the new 
model point; 

Figure 7 is a schematic representation of the model 
window in which the displayed model image shows the new 
model point and facets ; 

Figure 8 is a schematic representation of the model 
window showing the model image including the added model 
data, viewed from the same direction as the camera image 
of Figure 3; 

Figure 9 is a schematic flowchart showing the method 
steps for adding the new model data; 

Figure 10 is a further general illustration of the 
apparatus including a display screen; 



Figure 11 is a representation of a model window including 
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a display of a line representing the calculated 
trajectory; 

Figure 12 is a schematic representation of the addition 
of a new model point to an existing model according to a 
second aspect of the present invention; 

Figure 13A is a schematic representation of a camera 
selection window using camera icons; 

Figure 13B illustrates an alternative camera selection 
window using thumbnail icons; 

Figure 14 is a diagram illustrating the calculation of a 
ray intersecting a facet of the model; 

Figure 15 is a diagram illustrating the subdivision of a 
facet to include the added model point; 

Figure 16 is a diagram illustrating the display of a new 
model including the added point and new facets; 

Figure 17 is a flowchart illustrating the method 
described with reference to Figures 12 to 16; 

Figure 18 is a flowchart illustrating the step of 
replacing the existing facets with new facets using re- 
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triangulation ; 

Figure 19 is a diagram illustrating the identification of 
co-ordinates in a camera image of a feature corresponding 
to the added model point; 

Figure 20 is a diagrcim illustrating the calculation of 
the intersection with the facet of a ray through the 
camera image point and the added model point; 

Figure 21 is a flowchart illustrating the method 
described with reference to Figures 18 to 20; 

Figures 22 to 2 6 illustrate a third aspect of the present 
invention. Figure 22 illustrating schematically camera 
positions in relation to an object to be modelled;- 

Figure 23 illustrates a display screen of a computer 
interface allowing viewpoints to be selected by a user 
for selecting both camera image and model image; 

Figure 2 4 is a flowchart illustrating the method of 
implementing the interface of Figure 23; 

Figure 25 illustrates an alternative interface display 
allowing image selection using camera position icons; and 



Figure 26 is a flowchart illustrating the operafjion of 
the interface of Figure 25. 

Figure 1 schematically shows the components of a modular 
system in which the present invention may be embodied. 

These components can be effected as processor-implemented 
instructions^ hardware or a combination thereof. 

Referring to Figure 1^ the components are arranged to 
process data defining images (still or moving) of one or 
more objects in order to generate data defining a three- 
dimensional computer model of the object (s). 

The input image data may be received in a variety of 
ways, such as directly from one or more digital camieras, 
via a storage device such as a disk or CD ROM, by 
digitisation of photographs using a scanner, or by 
downloading image data from a database, for example via 
a datalink such as the Internet, etc. 

The generated 3D model data may be used to: display an 
image of the object(s) from a desired viewing position; 
control manufacturing equipment to manufacture a model of 
the object(s), for example by controlling cutting 
apparatus to cut material to the appropriate dimensions; 
perform processing to recognise the object(s), for 
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example by comparing it to data stored in a database; 
carry out processing to measure the object(s), . 
example by taking absolute measurements to record the 
size of the object(s), or by comparing the model with 
models of the object(s) previously generated to determine 
changes therebetween; carry out processing so as to 
control a robot to navigate around the object(s); store 
information in a geographic information system (GIS) or 
other topographic database; or transmit the object data 
representing the model to a remote processing device for 
any such processing, either on a storage device or as a 
signal (for example, the data may be transmitted . in 
virtual reality modelling language (VRML) format over the 
Internet, enabling it to be processed by a WWW browser); 
etc . 

The feature detection and matching module 2 is arranged 
to receive image data recorded by a still camera from 
different positions relative to the object(s) (the* 
different positions being achieved by moving the camera 
and/or the object(s)). The received data is then 
processed in order to match features within the different 
images (that is, to identify points in the images which 
correspond to the same physical point on the object(s)). 

The feature detection and tracking module 4 is arranged 
to receive image data recorded by a video camera as the 



relative, positions of the camera and object(S)- are 
changed (by moving the video camera and/or the 
object(s)). As in the feature detection and matching 
module 2, the feature detection and tracking module 4 
detects features, such as corners, in the images. 
However, the feature detection and tracking module 4 then 
tracks the detected features between frames of image data 
in order to determine the positions of the features in 
other images - 

The camera position calculation module 6 is arranged to 
use the features matched across images by the feature 
detection and matching module 2 or the feature detection 
and tracking module 4 to calculate the transformation 
between the camera positions at which the images were 
recorded and herice determine the orientation and position 
of the camera focal plane when each image was recorded. 

The feature detection and matching module 2 and the 
camera position calculation module 6 may be arranged to 
perform processing in an iterative manner. That is, 
using camera positions and orientations calculated by the 
camera position calculation module 6, the feature 
detection and matching module 2 may detect and match 
further features in the images using epipolar geometry in 
a conventional manner, and the further matched features 
may then be used by the camera position calculation 



module 6 to recalculate the camera position^, and 
orientations. 

If the positions at which the images were recorded are 
already known, then, as indicated by arrow 8 in Figure 1, 
the image data need not be processed by the feature 
detection and matching module 2, the feature detection 
and tracking module 4, or the camera position calculation 
module 6. For excimple, the images may be recorded by 
mounting a number of cameras on a calibrated rig arranged 
to hold the cameras in known positions relative to the 
ob ject ( s ) . 

Alternatively, it is possible to determine the positions 
of a plurality of cameras relative to the object(s) by 
adding calibration markers to the object (s) and 
calculating the positions of the cameras from the 
positions of the calibration markers in images recorded 
by the cameras* The calibration markers may comprise 
patterns of light projected onto the object(s). Camera 
calibration module 10 is therefore provided to receive 
image data from a plurality of cameras at fixed positions 
showing the object(s) together with calibration markers, 
and to process the data to determine the positions of the 
cameras . A preferred method of calculating the positions 
of the Cconeras (and also internal parameters of each 
camera, such as the focal length etc) is described in 
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"Calibrating and 3D Modelling with a Multi-Camera System" 
by Wiles and. Davison in 1999 IEEE Workshop on Multi-View 
Modelling -and Analysis of Visual Scenes , ; ISBN 0769501109 . 

The 3D object surface generation module 12 is arranged to 
receive image data showing the object(s) and data 
defining the positions at which the images were recorded, 
and to process the data to generate a 3D computer model 
representing the actual surface(s) of the object { s ) , such 
as a polygon mesh model. 

The texture data generation module 14 is arranged to 
generate texture data for rendering onto the surface 
model produced by the 3D object surface generation 
module 12. The texture data is generated from the input 
image data showing the object ( s ) . 

Techniques that can be used to perform the processing in 
the modules shown in Figure 1 are described in EP-A- 
0898245, EP-A-0901105, pending US applications 09/129077, 
09/129079 and 09/129080, the full contents of which are 
incorporated herein by cross-reference, and also Annex A. 

The present invention may be embodied in particular as 
part of the feature detection and matching module 2 
(although it has applicability in other applications, as 
will be described later) . 



Figure 10 illustrates generally the apparatus 100 of the 
present embodiment, comprising a processor 101, display 
monitor 102, and input devices including a computer mouse 
103 and keyboard 104. The mouse 103 enables signals such 
as an image point selection signal 112 (described below) 
to be input to the processor, 

A disc drive 105 also receives a floppy disc 106 carrying 
program code and/or image data for use by the processor 
101 in implementing the method steps of the present 
invention. 

The display monitor 102 has a display screen 107 which, 
in the present mode of operation of the program, displays 
a model window 108 and a ccimera image window 109- 

The processor 101 is connected to a modem 110 enabling 
program code or image data to be alternatively downloaded 
via the internet as an electronic signal 111. 

The method steps according to a first aspect of the 
present embodiment are illustrated in Figure 9 in which 
steps performed by the user and by the apparatus are 
separated by a broken line 9 0 representing the interface 
provided by the display screen 107 and input devices 
103,104. 
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The method begins from a starting- point at which the 
apparatus has already acquired ;a :set of existing model 
data.: derived -dEor example using the Vcomponents in Figure 
1 to process input image data in the form of a series of 
image frames obtained from a camera at respective 
different camera positions . The model data includes a 
set of model points and surface elements and estimates of 
the camera positions in the form of model co-ordinates 
for camera centres and look-directions derived for 
example by operation of camera position calculation 
module 6 to calculate camera positions based on the image 
data . 

At step 91, the apparatus displays in the display screen 
107 a model image 20 in the model window 10 8 as 
illustrated in Figure 2. Also displayed for side by side 
comparison is a camera image 3 0 in the camera image 
window 109 as illustrated in Figure 3. 

The model image 20 of Figure 2 is rendered using existing 
model data which the user wishes to update in order to 
add additional model data representing an additional 
feature 31 which is visible in the camera image of Figure 
3 but which has no equivalent in the model image 20 of 
Figure 2. The model image 20 and camera image 30 as 
shown in Figures 2 and 3 are generated as views from 
substantially the same viewing direction. 



At step 32, the user views the model image 20 and the 
camera image 30 and selects an image point 32 in the 
camera image 30 by using the computer mouse 103 to align 
a cursor 33 with the selected additional feature 31 and 
then clicking the mouse to generate an image point 
selection signal at step 93. 

At step 94, the apparatus receives the. image point 
selection signal and processes the signal to identify co- 
ordinates of the image point in the camera image 30- 

Since the camera image 30 is a two-dimensional projection 
of the object from which the model is derived, the two- 
dimensional co-ordinates obtained by user selection of 
the image point 3 2 do not specify uniquely a position in 
three dimensions at which the new model point is to be 
added. At step 95, the apparatus calculates the locus in 
three dimensions of the positions of possible model 
points corresponding to the selected image point 32 which 
are consistent with the geometric relationship between 
the object and the camera position from which the 
displayed camera image 3 0 was taken. This is illustrated 
in Figure 4 in which the model is viewed from a different 
viewpoint from that of Figure 2 and in which the locus is 
a straight line extending in the three dimensional space 
of the model from the model co-ordinates of the camera 
centre 40 and through the co-ordinates of the image point 
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3 2 in the camera image plane 41, 

An exemplary model point 42 lying on the locus 43 is 
illustrated in Figure 4 at one of the possible positions 
at which the new model point could be added. 

At step 96, the apparatus displays in the model window a 
new model image 21 as shown in Figure 5 in which a 
position indicator 50 lies on the locus 43 and is movable 
in response to movement of the computer mouse by the user 
so as to be constrained to follow a trajectory 51 
corresponding to the locus when projected into the plane 
of the model image 21. The new model image 21 of Figure 
5 is generated as a view of the model from a different 
viewpoint selected to clearly display the locus. Such 
different viewpoints are selected by the user by 
temporarily selecting a different mode of operation from 
a menu of available modes, the viewpoint selecting mode 
providing rotation of the model image in latitude and 
longitude in response to sideways and forward/reverse 
movement of the mouse respectively. 

At step 97, the user views the model image 21 and the 
position indicator 50 and decides upon an appropriate 
position of the position indicator 50 to represent the 
additional feature 31. At step 98, the user actuates the 
mouse to move the position indicator 50 to the selected 
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position, the apparatus updating the position of the 
position indicator appropriately at step 99, and at step 
910 the user clicks the mouse, thereby selecting . the 
desired position to set the position of the new model 
point. At step 911, the apparatus receives a selection 
input signal corresponding to the mouse click and freezes 
the position at which the position indicator 50 is 
displayed in the model image window. At .step 912, the 
apparatus determines the three-dimensional co-ordinates 
corresponding to the selected position of the additional 
model point, the co-ordinates being uniquely identified 
in three-dimensions from the known geometry of the locus 
and the selected position in the two-dimensional 
projection forming the model image 21 of Figure 5. 

At step 913, the apparatus adds the new model point to 
the existing model data and at step 914 displays the new 
model point 64 in the model window lOB together with 
existing model points, superimposed on the model image 2 0 
as shown in Figure 6. 

At step 915, the user views the model image and the new 
model point and selects a set of existing model points 
61, 6 2 and 63 for combining with the new model point 64 
to form a new subset of points to be used in the 
generation of surface elements of the model. The 
apparatus then generates the additional surface elements 
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shown as elements 70 and 71 in Figure 7 , Texture - data 
may then be rendered ontO; the resulting surface model 
using a texture : data generation module 14 as described 
above with reference to Figure 1. 

Figure 8 illustrates the model image incorporating the 
added model data when viewed from the same direction as 
the original camera image of Figure 3, In the model 
image of Figure 8, the ^additional feature 31 of the 
camera image 30 is represented by added model feature 80. 

The user may decide that the added model feature 80 does 
not adequately represent the additional feature 31 and, 
if so, may select an editing mode in which the position 
of the position indicator 50 may be adjusted and the 
resulting facetted model reviewed until the added model 
feature is judged to be correct, this further step 
requiring the input of further positioning signals and 
model point selecting signals responsive to user 
actuation of the mouse. 

In an alternative embodiment illustrated in Figure 11, 
the step 9 6 of displaying in the model window 108 the new 
model image 21 together with the indicator 50 may also 
include displaying a line 120 indicating the path of the 
tra j ectory 5 0 . 
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Alternative embodiments are envisaged in which for 
example non-linear locus calculation is effected, for 
example to take account of image distortion known to be 
present in the camera optics. Alternative means may be 
utilised for the input of data in place of a computer 
mouse, alternative forms of pointing device such as touch 
screen and touch pad devices being usable, or 
alternatively conventional keyboard devices may be used 
to input co-ordinates. 

In ' a further alternative embodiment, the step 915 in 
which the user selects existing model points for surface 
generation may be replaced by a step carried out by the 
apparatus to automatically select existing model points 
to be used in combination with the new model point as a 
basis for re-facetting the model - 

A preferred method of performing such automatic facetting 
will be described below. 

The method of the present invention can be implemented by 
a computer progrcim operating on the computer apparatus 
100, the program comprising processor implementable 
instructions for controlling the processor 101. The 
program may be stored in a storage medium such as floppy 
disk 106. An aspect of the present invention thus 
provides a storage medium storing processor implementable 
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instructions for carrying out the above described methods 

Further, the computer program may be obtained in 
electronic form for example by downloading the program 
code in the form of a signal 111 over a network such as 
the internet via the modem 38- 

Alternative embodiments of the present invention are 
envisaged in which for example the above described method 
and apparatus are used to process camera images obtained 
by selecting frames from a video camera recording, the 
frames representing different views of the object. The 
displayed images may additionally be modified to include 
dimensional information as a guide to the user in 
determining the optimum position of the new model point. 

A further aspect of the present embodiment will now be 
described, relating to the automatic re-facetting of the 
model when a new model point is added to a set of 
existing model points. Corresponding reference numerals 
to those of preceding figures will " be used where 
appropriate for corresponding elements. 

Figure 12 illustrates a new model point 64 which has been 
added to the data used to derive a model image 2 0 
displayed in a model window 108 in a display screen 107 
of a processor controlled apparatus 100 of the type 
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illustrated in Figure 10 and functioning as a system in 
the manner described above with reference to Figure 1- 

The addition of the new model point 64 may be the result 
of a process using selection of a camera image point and 
generating a locus in the model space as described above 
with reference to Figures 2 to 11 or may be the result of 
a different process, such as for excimple the input via a 
keyboard of numerals representing co-ordinates in the 
three-dimensional model space. 

In Figure 12, the model image 20 is representative of an 
irregularly shaped object represented schematically by a 
multi-facetted image in which the surface is comprised of 
a large number of triangular facets. In practice, the 
number of facets is likely to be greatly increased beyond 
the relatively small number illustrated in Figure 12 so 
that Figure 12 should therefore be regarded as schematic 
for the purpose of simplicity of representation in this 
respect. 

The method steps required to implement the method are 
illustrated in the flowchart of Figure 17 in which steps 
performed by the user are illustrated in the left-hand 
portion of the flowchart, steps implemented by the 
apparatus are shown in the right-hand portion of the 
flowchart and an interface between the user and the 
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apparatus is represented as a broken- line 90> . In 
practice, the interface is comprised of the display 
screen :107 and; the computerl mouse 106 allowing the input 
of pointing signals in con junction; with the display of a 
cursor 33 on the display screen 107. 

The following method steps illustrated in Figure 17 will 
be described with reference to Figures 12 to 16. At step 
170, the user selects via mode icons 230 a mode of 
operation of the apparatus for choosing a view of the 
model and the apparatus responds by displaying the model 
image 20 in the model image window 108. The user 
actuates the mouse 10 3 to orient the model view to a 
position which is judged to be appropriate. 

At step 171, the user selects a mode of operation for the 
addition of model points and the apparatus responds by 
displaying a prompt for the input of the model point 
information. The user inputs co-ordinates of the added 
model point and, at step 172, the apparatus displays the 
new model point in ~ the- model image window 10 8 as 
illustrated in Figure 12. The apparatus also displays on 
the display screen 107 a camera selection window 130 as 
illustrated in Figure 13A in which the camera positions 
relative to the object represented by the model image are 
graphically represented in a manner which enables the 
user to choose one of the cameras as being appropriately 
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located for the purpose of defining a centre of 
projection to allow the new model point 64 to be 
projected onto the existing model. The user may for 
example already have knowledge of the object being 
modelled and a general indication of the required camera 
view. 

In the camera selection window 130, the cameras are 
represented at their positions relative to a 
representation of the object 131 by respective camera 
icons 132 such that the user is able to select one of the 
cameras by use of the mouse, the user aligning the cursor 
33 onto a selected one of the camera icons and clicking 
the mouse 103 to effect selection. 

At step 174, the apparatus receives the camera selecting 
signal and determines the position of the camera centre 
147 in the three-dimensional co-ordinate system of the 
model . 

At step 175, the apparatus calculates the manner in which 
the new model point 64 is projected onto the surface of 
the model by calculating a ray in the model space through 
the position of the ccimera centre and the co-ordinates of 
the new model point- As shown in Figure 14, a ray 140 
defined in the above manner intersects the surface of the 
model at a point of intersection 141 which lies within a 
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facet 142 defined by apices 143, 144 and 145 and. also 
intersects a second facet 14 6 on exiting the model 
surface . 

At step 176, the apparatus replaces the existing facet 
142 with new facets 150, 151 and 152 as illustrated in 
Figure 15, each of which includes the new model point 64 
as a respective apex. At step 177, the apparatus 
displays the new model image including the added point 64 
and the new facets 150, 151 and 152 as illustrated in 
Figure 16 in which the new facets are highlighted by 
being cross-hatched (facet 152 is hidden from view). 

Step 176 of replacing the existing facet with new facets 
is illustrated in greater detail in the flowchart of 
Figure 18. At step 180, the apparatus determines whether 
the ray 140 intersect one of the model facets. If no 
intersection occurs, the apparatus displays a prompt to 
the user to select a model facet at step 181 and at step 
182 the user responds by selecting a facet to be 
replaced, selection being carried out using the mouse and 
cursor- At step 183, the apparatus determines the set of 
co-ordinates upon which the selected facet is based and, 
at step 184, adds the new model point to this set of co- 
ordinates. In this example, since the facet being 
replaced is triangular, the set of co-ordinates on which 
the facet is based consists of three model points. When 



the new model point is added, there are four model points 
as a basis for re-triangulation . At step 185, the 
apparatus performs re-triangulation to -define three 
triangular, facets which connect the set of four points to 
form part of the surface of the model as illustrated in 
Figure 15. 

If at step 180, the apparatus determines that the ray 
does in fact intersect a model facet 14 2 as shown in 
Figure 14, the point of intersection 141 is determined, 
thereby defining the facet 142 which is intersected by 
the ray, and the set of co-ordinates of the intersected 
facet are then used in combination with the new model 
point at step 184 to define the set of new co-ordinates. 
If, as in the case of Figure 14, more than one facet is 
intersected by the ray 140, the apparatus determines at 
step 185 which of the facets is closest to the new model 
point 64 as a subject for re-triangulation . In the 
example of Figure 14, the facet 142 is therefore selected 
in preference to facet 146 since it is closer to the new 
model point 64. 

Figure 13B illustrates an alternative method of selecting 
the camera position by using a camera selection window 
130 which includes a series of thumbnail icons 133, each 
thumbnail icon comprising a thumbnail image derived from 
the image data obtained from a respective camera 
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position. The user may thereby select from the displayed 
thumbnail images the appropriate camera position for 
viewing: the required aspect of the object represented by 
the model image and by clicking the mouse 103 when cursor 
5 33 is on the thumbnail icon 133, generates a pointing 
signal 112 received at step 174 of Figure 17 by the 
apparatus, thereby enabling the required camera position 
to be determined as a centre of projection. 

10 In the above described example, the centre of projection 
for projecting the new model point onto the surface of 
the model is defined as being the centre of the camera. 
The centre of projection may alternatively be defined in 
terms of the point in the jLmage plane of the camera 

15 corresponding to the location of the image point 
corresponding to the new model point. For example, in 
Figure 19, a Cconera image 30 is displayed in a Cconera 
image window 190 to allow the user to select a camera 
image point 191 determined by the user to correspond to 

20 the new model point 64. As illustrated in the flowchart 
of Figure 21, the co-ordinates of the Ccimera image point 
are input at step 210 to enable the apparatus to 
calculate at step 211 the ray in the model space through 
the co-ordinates of the added model point and camera 

2 5 image point as illustrated in Figure 2 0 where the 
position 200 of the camera image point in the camera 
plane 201 is used to determine the trajectory of the ray 
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Alternative devices may be used in place of the computer 
mouse 103 for the input of selection signals, including 
5 for example any conventional pointing device such as a 
touch screen or touch pad device* Alternatively^ a 
keyboard 104 may be used for the input of commands or co- 
ordinates. 

10 In the method of Figure 17, the user may choose to change 
from one mode to another at any time by selecting one of 
the mode icons 2 30. 

The method of the above aspect of the present invention 
15 described with reference to Figures 1, 10, and 12 to 20 
can be implemented by a computer program operating on the 
computer apparatus 100, the progrsuna comprising processor 
implementable instructions for controlling the processor 
101. The program may be stored in a storage medium such 
20 as floppy disk 106. An aspect of the present invention 
thus provides a storage medium storing processor 
implementable instructions for carrying out the above 
described method . 
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Further, the computer program may be obtained in 
electronic form for example by downloading the progrcon 
code as a signal 111 over a network such as the internet 
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via the modem 110- 

A further aspect of the present embodiment: will now be 
described using corresponding reference numerals to those 
of preceding figures where appropriate for corresponding 
elements. This aspect of the embodiment relates to the 
provision of a method and apparatus enabling an interface 
to allow a user to evaluate the quality of a model of the 
type discussed above, and in particular of the type 
discussed with reference to Figure 1 using the apparatus 
described above with reference to Figure 10. 

As previously discussed, a user may adopt one of a number 
of techniques for refining and editing model data in 
order to achieve an improved model image. In order to 
evaluate the quality of the model image, this aspect of 
the embodiment allows views of the model image and camera 
image to be presented in respective model image windows 
and camera image windows on the display screen and for 
the respective images to be presented such that both the 
camera image and model image represent views of the 
object from substantially the Scime viewpoint and in 
respect of which substantially the same image settings 
such as magnification, field of view, etc, are provided 
(these latter parameters are referred to below as "camera 
intrinsics " ) . 



Figure 22 illustrates the relationship between a physical 
object 220 which is the subject of the modelling exercise 
and a set of camera positions L(i), relative to the 
object 220/ from which a set of frames of image data are 
5 obtained^ a corresponding camera image I(i) being 
obtained. The camera images may be obtained by moving a 
single camera successively into the camera positions 
Ii(i), by having a .set of different cameras or by moving 
the object relative to a stationary camera, for example. 

10 

Having obtained model data allowing model images to be 
displayed, the user wishes to evaluate the model by 
displaying side by side a camera image and a model image - 
In Figure 22, camera position L(3) is of particular 
15 interest to the user. 

Using the apparatus of Figure 10, the user operates the 
apparatus to achieve this result using the method steps 
illustrated in the flowchart of Figure 24 which will be 
20 illustrated below with reference to Figure 23. 

At step 240, the user selectis the required mode of 
operation for displaying camera and model images for the 
purpose of evaluation, mode selection being achieved 
25 using the interface provided by the display screen 107, 
the cursor 33 and the mouse 103 to select one of the mode 
icons 2 30 located in a peripheral region of the display 
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screen as shown in Figure 23. 

At step 2 41, the apparatus generates camera image data 
for each of the frames of image data, using the thumbnail 
5 image format, and displays the thumbnail images as icons 
231 within an icon window 232 of the display screen 107. 

The icons 231 are displayed in a sequence as calculated 
by ccimera position calculation module 6 which corresponds 

10 to the spatial relationship of the positions L(i) as 
shown in Figure 22, so that the sequence Ii(i), i = 1 to 
n progressing from left to right is maintained in the 
layout of the icons on the display screen 107 such that 
images I(i), i = 1 to n, are positioned from left to 

15 right according to the value of i. 

For simplicity of representation, the images shown in 
Figure 23 are those of a regular polyhedron in which an 
X is drawn on one of the faces so that the apparent 
20 position of the x in each of the displayed thumbnail 
images corresponds to- the view which would be obtained 
from the camera positions L(i) . 

At step 242 the user views the icons and at step 243 the 
25 user selects one of the icons as being of particular 
relevance for the purpose of evaluation of the images. 
The user selects the icon as indicated in Figure 23 by 
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the cursor 33 overlaying the third image, i. .= 3, 
corresponding to selection of the camera position L(3) of 
Figure 22. 

At step 244, the apparatus receives the icon selection 
input and at step 245, the apparatus identifies the 
selected camera image for display in a camera image 
window 109. At step 246, the apparatus determines the 
position data for the selected camera by accessing data 
stored with the camera image data and at step 247 
calculates the model image data using the selected 
position data to define the viewpoint for the model. In 
calculating the model image data, the apparatus also uses 
camera intrinsic parameters stored with the camera image 
data. The intrinsic parameters of the camera comprise 
the focal length, the pixel aspect ratio, the first order 
radial distortion coefficient, the skew angle (between 
the axes of the pixel grid) and the principal point (at 
which the Ccimera optical axis intersects the viewing 
plane ) . 

At step 248, the apparatus displays a model image 20 in 
the model image window 108 and the camera image 3 0 in a 
camera image window 109, thereby allowing the user to 
view and compare the selected camera image and the model 
image as calculated from a corresponding viewpoint. 
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In Figure 23, the icons 231 are linked in series by links 
233. If necessary, a large number of such icons may be 
displayed in an array comprising a number of rows, 
maintaining the links between successive icons in order 
5 to visually indicate the continuity of the sequence (i.e. 
the direction of increasing i). The use of such links 
therefore assists in providing the user with an 
indication of where the most appropriate image is to be 
selected. 

10 

After viewing the images for a selected viewpoint, the 
user may then choose to view camera and model images for 
different viewpoints by selecting different icons, 
repeating step 243 of Figure 24, and resulting in the 
15 apparatus repeating steps 244 to 248 to enable the 
further views to be seen. 

If the user then decides that the model data requires 
editing, the user may then select a different mode of 
20 operation by selecting the appropriate mode icon 230 for 
further operation of the apparatus . 

An alternative embodiment will now be described with 
reference to Figure 25 and the flowchart of Figure 26. 
25 Referring to Figure 26, at step 260 the user selects a 
required mode of operation by selecting the appropriate 
mode icon 230 of Figure 25. The apparatus responds by 
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generating and displaying icons 250 in a camera position 
window 251. 

Within the camera position window 251, a display 
generated by the apparatus at step 261 comprises a 
representation 252 of the object based upon the model 
data together with representations of cameras at 
positions L(i), i = 1 to n, such that the relative 
positions of the cameras and the representation 252 
correspond to the calculated camera positions developed 
by the camera position calculation module 6 of Figure 1. 
The representation 252 is thereby placed at the origin of 
the co-ordinate system of the model and the icons 250 
located in effect at the calculated camera positions. 

This representation of the relative positions of the 
cameras and object allows the user to easily select a 
viewing point for the camera and model images to be 
displayed. In order to select a particular viewpoint, 
the user at step 262 views the icons 250 within the 
window 251 and at step 263 selects one of the icons at 
the desired cetmera position. The apparatus responds at 
step 2 65 by identifying the camera image data 
corresponding to the selected camera position. At step 
266, the apparatus then proceeds to calculate the model 
image data using the selected position data as a 
viewpoint and using camera intrinsic parameters stored in 
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conjunction with the camera image data identified in step 
265. 

At step 267, the apparatus then displays the model image 
in model image window 10 8 and the camera image 30 in 
camera image window 109 to be viewed by the user at step 
2 68. The user is then able to evaluate the quality of 
the image by comparison between the images. 

In each of the display interfaces of Figures 23 and 25, 
the camera image window 10 9 and the model image window 
10 8 may be moved relative to one another using a drag and 
drop method by means of actuating the mouse. Similarly, 
the icon windows 23 2 and 251 may be moved relative to the 
image windows 108 and 109, thereby allowing the user to 
arrange the windows for maximum ease of selection and 
comparison . 

The method of the present invention can be implemented by 
a computer program operating on the computer apparatus 
100, the program comprising processor implementable 
instructions for controlling the processor 101. The 
program may be stored in a storage medium such as floppy 
disk 106. An aspect of the present invention thus 
provides a storage medium storing processor implementable 
instructions for carrying out the above described method. 



Further, the computer program may be obtained in 
electronic form for example by downloading the program 
code as a. signal 111 over a network such as the internet 
via the modem 110. 

The present application incorporates by cross-reference 
the full contents of the applicant's following co-pending 
applications: 9927876.4 (attorney ref : 2635901); 

9927875.6 (attorney ref : 2636501); 9927906.9 (attorney 
ref: 2641601); 9927907.7 (attorney ref: 2641701); 
9927909.3 (attorney ref : 2641901); 9929957.0 (attorney 
ref: 2641001); 0001300.3 (attorney ref: 2636001) and 
0018492.9 (attorney ref: 2635801). 




40 263620X 



ANNEX A 



1. CORNER DETECTION 

5 1.1 Summary 

This process described below calculates corner points, to 
sub-pixel accuracy, from a single grey scale or colour 
image. It does this by first detecting edge boundaries in 
10 the image and then choosing corner points to be points 
where a strong edge changes direction rapidly. The 
method is based on the facet model of corner detection, 
described in Haralick and Shapiro^. 

15 1.2 Algorithm 

The algorithm has four stages: 

(1) Create grey scale image (if necessary); 

2 0 (2) Calculate edge strengths and directions; 

(3) Calculate edge boundaries ; 

(4) Calculate corner points. 



1-2-1 Create grey scale image 



25 



The corner detection method works on grey scale images. 
For colour images, the colour values are first converted 
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to floating point grey scale values using the formula: 

grey_scale = (0.3 x red) +(0.59 x green) +(0.11 x blue) 

A-1 

5 

This is the standard definition of brightness as defined 
by NTSC and described in Foley and van Dam^^. 

10 1.2,2 Calculate edge strengths and directions 

The edge strengths and directions are calculated using 
the 7x7 integrated directional derivative gradient 
operator discussed in section 8.9 of Haralick and 
15 Shapiro^. 

The row and column forms of the derivative operator are 
both applied to each pixel in the grey scale image. The 
results are combined in the standard way to calculate the 
20 edge strength and edge direction at each pixel. 

The output of this part of the algorithm is a complete 
derivative image . 
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1.2.3 Calculate edge boundaries 
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The edge boundaries are calculated by using a zero 
crossing edge detection method based on a set of 5x5 
kernels describing a bivariate cubic, fit to the 
neighbourhood of each pixel - 

5 

The edge boundary detection method places an edge at all 
pixels which are close to a negatively sloped zero 
crossing of the second directional derivative taken in 
the direction of the gradient, where the derivatives are 
10 defined using the bivariate cubic fit to the grey level 
surface. The subpixel location of the zero crossing is 
also stored along with the pixel location. 

The method of edge boundary detection is described in 
15 more detail in section 8.8.4 of Haralick and Shapiro^. 

1.2.4 Calculate corner points 

The corner points are calculated using a method which 
2 0 uses the edge boundaries calculated in the previous 
step . 
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Corners are associated with two conditions: 

(1) the occurrence of an edge boundary; and 

(2) significant changes in edge direction. 

Each of the pixels on the edge boundary is tested for 
"cornerness" by considering two points equidistant to it 
along the tangent direction. If the change in the edge 
direction is greater than a given threshold then the 
point is labelled as a corner. This step is described in 
section 8,10.1 of Haralick and Shapiro^. 

Finally the corners are sorted on the product of the edge 
strength magnitude and the change of edge direction. The 
top 200 corners which are separated by at least 5 pixels 
are output . 

2 - FEATURE TRACKING 



2 - 1 Summary 



This process described, below tracks feature points 
(typically corners) across a, sequence of grey scale or 
colour images. 



The tracking method uses a constant image velocity Kalman 
filter to predict the motion of the corners, and a 
correlation based matcher to make the measurements of 
corner correspondences . 

The method assumes that the motion of corners is smooth 
enough across the sequence of input images that a 
constant velocity Kalman filter is useful, and that 
corner measurements and motion can be modelled by 
gaussians. 

2 . 2 Algorithm 

1) Input corners from an image. 

2) Predict forward using Kalman filter. 

\, 

3) If the position uncertainty of the predicted corner 



is greater than a threshold. A, as measured by the 
state positional variance, drop the corner from the 
list of currently tracked corners. 

4) Input a new image from the sequence. 

5) For each of the currently tracked corners: 

a) search a window in the new image for pixels 
which match the corner; 

b) update the corresponding Kalman filter, using 
any new observations ( i . e . matches ) - 

6) Input the corners from the new image as new points 
to be tracked (first, filtering them to remove any 
which are too close to existing tracked points). 

7) Go back to (2) 
2.2.1 Prediction 



This uses the following standard Kalman filter equations 
for prediction, assuming a constant velocity and random 
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uniform gaussian acceleration model for the dynamics: 

X = G , X • ■ A-2 



nt-1 n*l, n n n + l,n n 



where x is the 4D state of the system, (defined by the 
5 position and velocity vector of the corner), K is the 
state covariance matrix, G is the transition matrix, and 
Q is the process covariance matrix. 

In this model, the transition matrix and process 
10 covariance matrix are constant and have the following 
values: 



n^l.n I 0 jJ . 



0 oll^ 



.A-5 
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2.2-2 Searching and matching 
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This uses the positional uncertainty (given by the top 
two diagonal elements of the state covariance matrix, K) 
to define a region in which to search for new 
measurements (i.e. a range gate). 

The range gate is a rectangular region of dimensions: 
Ax = yx^, Ay = ^ A-6 

The correlation score between a window around the 
previously measured corner and each of the pixels in the 
range gate is calculated. 

The two top correlation scores are kept. 

If the top correlation score is larger than a threshold, 
Co/ and the difference between the two top correlation 
scores is larger than a threshold AC, then the pixel with 
the top correlation score is kept as the latest 
measurement . 



2.2-3 Update 
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The measurement is used to update the . Kalman filter in 

the standard way: 

G KH'^{HKH^+R)'^ ^""^ 

x-x+Gix-Hx) A-8 

{I-GH) K -^-9 

where G is the Kalman gain, H is the measurement matrix, 
and R is the measurement covariance matrix. 

In this implementation, the measurement matrix and 

measurement covariance matrix are both constant, being 
given by : 

//-(JO) A-10 

R = a^j . • - .A-11 



2.2.4 Parameters 

The parameters of the algorithm are: 

Initial conditions: Xq and Kq- 
Process velocity variance: a^^. 
Measurement variance: . 
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Position uncertainty threshold for loss of 
track: A- 

Covariance threshold: Cq- 
Matching ambiguity threshold: AC. 

For the initial conditions, the position of the first 
corner measurement and zero velocity are used, with an 
initial covariance matrix of the form: 



^0 = 



0 0 
0 ^l^; 



Oq^ is set to Oq^ = 200 (pixels/frame) 2- 

The algorithm's behaviour over a long sequence is anyway 
not too dependent on the initial conditions • 



The process velocity variance is set to the fixed value 
of 50 (pixels/f rame)2. The process velocity variance 
would have to be increased above this for a hand-held 
sequence. In fact it is straightforward to obtain a 
reasonable value for the process velocity variance 
adaptively. 



The measurement 
model: 

where K = ^/(KllK22) is a measure of the positional 
uncertainty, r is a parsimeter related to the likelihood 
of obtaining an outlier, and a is a parameter related to 
the measurement uncertainty of inliers, "r" and "a" are 
set to r=0 . 1 and a=1.0. 

This model takes into account, in a heuristic way, the 
fact that it is more likely that an outlier will be 
obtained if the range gate is large - 

The measurement variance (in fact the full measurement 
covariance matrix R) could also be obtained from the 
behaviour of the auto-correlation in the neighbourhood of 
the measurement. However this would not take into 
account the likelihood of obtaining an outlier. 

The remaining parameters are set to the values: A=400 
pixels^, Co=0 . 9 and AC=0.001. 
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variance is obtained from the following 



(rK+a) ^-13 
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3 . 3D SURFACE GENERATION 
3 . 1 Architecture 

5 

In the method described below it is assumed that the 
object can be segmented from the background in a set of 
images completely surrounding the object- Although this 
restricts the generality of the method, this constraint 
10 can often be arranged in practice, particularly for small 
objects. 

The method consists of five processes, which are run 
consecutively: 

15 

- First, for all the images in which the camera 
positions and orientations have been calculated, 
the object is segmented from the background, using 
colour information. This produces a set of binary 
20 images, where the pixels are marked as being either 

object or background. 
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- The segmentations are used, together with the 
camera positions and orientations, to generate a 
voxel carving, consisting of a 3D grid of voxels 
enclosing the object. Each of the voxels is marked 

5 as being either object or empty space. 

- The voxel carving is turned into a 3D surface 
triangulation, using a standard triangulation 
algorithm (marching cubes). 

10 

The number of triangles is reduced substantially by 
passing the triangulation through a decimation 
process. 

15 - Finally the triangulation is textured, using 
appropriate parts of the original images to provide 
the texturing on the triangles. 



3 . 2 Segmentation 
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The aim of this process is to segment an object (in front 
of a reasonably homogeneous coloured background) in an 
image using colour information. The resulting binary 
image is used in voxel carving. 

Two alternative methods are used: 

Method 1: input a single RGB colour value 
representing the background colour - each RGB pixel 
in the image is examined and if the Euclidean 
distance to the background colour (in RGB space) is 
less than a specified threshold the pixel is 
labelled as background (BLACK) . 

Method 2: input a "blue" image containing a 
representative region of the background. 

The algorithm has two stages: 

(1) Build a hash table of quantised background colours 



(2) Use the table to segment each image- 
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Step 1) Build hash table 

Go through each RGB pixel, p, in the "blue" background 
image . 

Set q to be a quantised version of p. Explicitly: 

q = (p+t/2) /t A-14 

where t is a threshold determining how near RGB values 
need to be to background colours to be labelled as 
background - 

The quantisation step has two effects: 

1) reducing the number of RGB pixel values, thus 
increasing the efficiency of hashing; 

2 ) defining the threshold for how close a RGB pixel 
has to be to a background colour pixel to be 
labelled as background . 



q is now added to a hash table (if not already in the 
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table) using the (integer) hashing function 

h{q) = (c?_red & 7) *2^6+ (qr__green. & 7) '2^3+ (g_biue fit 7) 

.A-15 

That is, the 3 least significant bits of each colour 
field are used. This function is chosen to try and 
spread out the data into the available bins. Ideally 
each bin in the hash table has a small number of colour 
entries- Each quantised colour RGB triple is only added 
once to the table (the frequency of a value is 
irrelevant ) . 

Step 2 ) Segment each image 

Go through each RGB pixels v, in each image. 

Set w to be the quantised version of v as before. 

To decide whether w is in the hash table, explicitly look 
at all the entries in the bin with index h(w) and see if 
any of them are the same as w. If yes, then v is a 
background pixel - set the corresponding pixel in the 
output image to BLACK. If no then v is a foreground 



pixel - set the corresponding pixel in the output^ image 
to WHITE 

Post Processing: For both methods a post process is 
performed to fill small holes and remove small isolated 
regions. 

A median filter is used with a circular window. (A 
circular window is chosen to avoid biasing the result in 
the X or y directions). 

Build a circular mask of radius r. Explicitly store the 
start and end values for each scan line on the circle. 

Go through each pixel in the binary image. 

Place the centre of the mask on the current pixel. Count 
the number of BLACK pixels and the number of WHITE pixels 
in the circular region. 

If (#WHITE pixels ^ #BIjACK pixels) then set corresponding 
output pixel to WHITE. Otherwise output pixel is BLACK. 
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3 . 3 Voxel carving 

The aim of this process is to produce a 3D voxel grid^ 
enclosing the object^ with each of the voxels marked as 
either object or empty space. 

The input to the algorithm is: 

a set of binary segmentation images, each of which 
is associated with a camera position and 
orientation; 

2 sets of 3D co-ordinates, (xmin, ymin, zmin) and 
(xmax, ymax, zmax), describing the opposite 
vertices of a cube surrounding the object; 

a pareunaeter, n, giving the number of voxels 
required in the voxel grid. 

A pre-processing step calculates a suitable size for the 
voxels (they are cubes) and the 3D locations of the 
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using (xmin, ymin, zmin) and (xmax^^ ymax. 

Then, for each of the voxels in the grid, the mid-point 
5 of the voxel cube is projected into each of the 
segmentation images. If the projected point falls onto 
a pixel which is marked as background, on any of the 
images, then the corresponding voxel is marked as empty 
space, otherwise it is marked as belonging to the object. 

10 

Voxel carving is described further in "Rapid Octree 
Construction from Image Sequences" by R. Szeliski in 
CVGIP: Image Understanding, Volume 58, Number 1, July 
1993, pages 23-32. 

15 

3 . 4 Marching cubes 

The aim of the process is to produce a surface 
triangulation from a set of samples of an implicit 
20 function representing the surface (for instance a signed 
distance function) . In the case where the implicit 
function has been obtained from a voxel carve, the 



voxels , 
zmax) . 



implicit, function takes the value -1 for samples^ which 
are inside the object and +1 for samples which are 
outside the object. 

Marching cubes is an algorithm that takes a set of 
saunples of an implicit surface (e.g. a signed distance 
function) sampled at regular intervals on a voxel grid, 
and extracts a triangulated surface mesh. Lorensen and 
Cline^^^ and Bloomenthal^^ give details on the algorithm 
and its implementation. 

The marching-cubes algorithm constructs a surface mesh by 
"marching" around the cubes while following the zero 
crossings of the implicit surface f(x)=0, adding to the 
triangulation as it goes. The signed distance allows the 
marching-cubes algorithm to interpolate the location of 
the surface with higher accuracy than the resolution of 
the volume grid. The marching cubes algorithm can be 
used as a continuation method (i.e. it finds an initial 
surface point and extends the surface from this point) . 



3 . 5 Decimation 
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The aim of the process is to reduce the number of 
triangles in the model, making the model more compact and 
therefore easier to load and render in real time. 

5 The process reads in a triangular mesh and then randomly 
removes each vertex to see if the vertex contributes to 
the shape of the surface or not.^ (i.e. if the hole is 
filled, is the vertex a "long" way from the filled hole). 
Vertices which do not contribute to - the shape are kept 
10 out of the triangulation. This results in fewer vertices 
(and hence triangles) in the final model. 

The algorithm is described below in pseudo-code. 

15 INPUT 

Read in vertices 

Read in triples of vertex IDs making up triangles 

PROCESSING 
2 0 Repeat NVERTEX times 

Choose a random vertex, V, which hasn't been 
chosen before 

Locate set of all triangles having V as a 
vertex J. S 

2 5 Order S so adjacent triangles are next to each 
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other 

Re-trlangulate triangle set^ ignoring V (i.e. 
remove selected triangles & V and then fill in 
hole) 

5 Find the maximum distance between V and the 

plane of each triangle 
If (distance < threshold) 

Discard V and keep new triangulation 

Else 

10 Keep V and return to old triangulation 

OUTPUT 

Output list of kept vertices 
Output updated list of triangles 

The process therefore combines adjacent triangles in the 
model produced by the marching cubes algorithm, if this 
can be done without introducing large errors into the 
model. 

20 

The selection of the vertices is carried out in a random 
order in order to avoid the effect of gradually eroding 
a large part of the surface by consecutively removing 
neighbouring vertices . 

25 

3 . 6 Further Surface Generation Techniques 



% 
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Further techniques which may be employed to generate a 3D 
computer model of an object ; surface include voxel 
colouring, for example as described in "Photorealistic 
Scene Reconstruction by Voxel Coloring" by Seitz and Dyer 
5 in Proc. Conf. Computer Vision and Pattern Recognition 
1997, pl067-1073, "Plenoptic Image Editing" by Seitz and 
Kutulakos in Proc. 6th International Conference on 
Computer Vision, pp 17-24, "What Do N Photographs Tell Us 
About 3D Shape?" by Kutulakos and Seitz in University of 
10 Rochester Computer Sciences Technical Report 680, January 
199 8, and "A Theory of Shape by Space Carving" by 
Kutulakos and Seitz in University of Rochester Computer 
Sciences Technical Report 692, May 1998. 

15 4. TEXTURING 

The aim of the process is to texture each surface polygon 
(typically a triangle) with the most appropriate image 
texture. The output of the process is a VRML model of 
2 0 the surface, complete with texture co-ordinates. 

The triangle having the largest projected area is a good 
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triangle to use for texturing, as it is the triangle for 
which the texture will appear at highest resolution. 



A good approximation to the triangle with the largest 
projected area, under the assumption that there is no 
substantial difference in scale between the different 
images, can be obtained in the following way. 

For each surface triangle, the image "i" is found such 
that the triangle is the most front facing (i.e. having 
the greatest value for fi^Vi, where is the triangle 
normal and v^ is the viewing direction for the "i" th 
camera) . The vertices of the projected triangle are then 
used as texture co-ordinates in the resulting VRML model . 

This technique can fail where there is a substantial 
amount of self -occlusion, or several objects occluding 
each other. This is because the technique does not take 
into account the fact that the object may occlude the 
selected triangle. However, in practice this does not 
appear to be much of a problem. 
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It has been found that, if every image is used for 
texturing then this can result in very large VRML models 
being produced. These can be cumbersome to load and 
render in real time. Therefore, in practice, a subset of 
images is used to texture the model- This subset may be 
specified in a configuration file. 
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CLAIMS 

1 . A method of operating an apparatus for generating 
model data representative of a model in a three 
dimensional space from image data representative of a set 
of camera images of an object; 

the apparatus performing the steps of; 

displaying a model image based on an existing set of 
model data; 

displaying one of the camera images of the object 
for selection by a user of an additional feature to be 
represented by additional model data; 

receiving an image point selection signal responsive 
to user actuation of an input means and identifying co- 
ordinates of an image point in the camera image defining 
the selected additional feature; 

calculating a locus in the three dimensional space 
defining positions of possible model points corresponding 
to the image point and consistent with the geometric 
relationship between the object and a camera position 
from which the displayed camera image was taken; 

displaying a position indicator in the model image 
at co-ordinates in the model image corresponding to one 
of the possible model points on the locus; 

receiving positioning signals responsive to user 
actuation of the input means and updating the co- 
ordinates of the position indicator such that movement of 
the position indicator is constrained to follow a 
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trajectory in the model image corresponding to the locus; 

receiving a model point selecting signal responsive 
to user actuation of the input means and determining 
selected co-ordinates of the position indicator to be the 
position indicator co-ordinates at the time of receiving 
the model point selecting signal; and 

determining co-ordinates of the additional model 
point in the three dimensional space corresponding to the 
selected co-ordinates of the position indicator. 

2. A method as claimed in claim 1 including displaying 
in the model image a line representing the locus. 

3. A method as claimed in any preceding claim wherein 
the locus is a straight line in the three dimensional 
space . 

4. A method as claimed in any preceding claim wherein 
the input means comprises a computer mouse and wherein 
said positioning signals are responsive to user actuation 
of the mouse by clicking the mouse during movement and 
releasing the mouse at a selected position to generate 
the model point selecting signal. 

5. A method as claimed in any preceding claim including 
the step of displaying a symbol representative of the 
additional model point at a model image point 
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corresponding to the selected co-ordinates. 

6. A method as claimed in claim 5 comprising the 
further step of editing the position of the additional 
model point in response to receiving further positioning 
signals and model point selecting signals responsive to 
user actuation of the input means. 

7 . A method as claimed in any preceding claim including 
the further step of receiving a processing instruction 
signal and, responsive to said signal, implementing a 
model generating process to incorporate the additional 
model point into the model data. 

8 . A method as claimed in claim 7 including the step of 
generating surface elements of the model from the model 
data including the additional model point and displaying 
said surface elements in the model image. 

9. Apparatus for generating model data representative 
of a model in a three dimensional space from image data 
representative of a set of cjamera images of an object; 

the apparatus comprising; 

an interface comprising display means operable to 
display images to a user and input means responsive to 
user actuation; 

control means operable to control the display means 
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to display a model image based on an existing set of 
model data and to display one of the camera images of the 
object for selection by a user of an additional feature 
to be represented by additional model data; 

receiving means for receiving an image point 
selection signal responsive to user actuation of the 
input means and identifying co-ordinates of an image 
point in the camera image defining the selected 
additional feature ; 

calculating means for calculating a locus in the 
three dimensional space defining positions of possible 
model points corresponding to the image point and 
consistent with the geometric relationship between the 
object and a camera position from which the displayed 
camera image was taken; 

the control means being further operable to control 
the display means to display a position indicator in the 
model image at co-ordinates in the model image 
corresponding to one of the possible model points on the 
locus; 

the apparatus further comprising means for receiving 
positioning signals responsive to user actuation of the 
input means and updating the co-ordinates of the position 
indicator such that movement of the position indicator is 
constrained to follow a trajectory in the model image 
corresponding to the locus; 

means for receiving a model point selecting signal 
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responsive to user actuation of the input means and 
determining selected co-ordinates of the position 
indicator to be the position indicator co-ordinates at 
the time of receiving the model point selecting signal; 
and 

means for determining co-ordinates of the additional 
model point in the three dimensional space corresponding 
to the selected co-ordinates of the position indicator. 

10. Apparatus as claimed in claim 9 wherein the control 
means is operable to control the display means to display 
in the model image a line representing the locus. 

11. Apparatus as claimed in any of claims 9 and 10 
wherein the calculating means is operable to calculate 
the locus as a straight line in the three dimensional 
space . 

12. Apparatus as claimed in any of claims 9 to 11 
wherein the input means comprises a computer mouse and 
wherein said positioning signals are responsive to user 
actuation of the mouse by clicking the mouse during 
movement and releasing the mouse at a selected position 
to generate the model point selecting signal. 

13. Apparatus as claimed in any of claims 9 to 12 
wherein the control means is operable to control the 
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display means to display a symbol representative of the 
additional model point at a model image point 
corresponding to the selected co-ordinates. 

14. Apparatus as claimed in claim 13 comprising editing 
means for editing the position of the additional model 
point in response to receiving further positioning 
signals and model point selecting signals responsive to 
user actuation of the input means. 

15. Apparatus as claimed in any of claims 9 to 14 
including model generating means operable to receive a 
processing instruction signal and, responsive to said 
signal, to implement a model generating process to 
incorporate the additional model point into the model 
data- 

16. Apparatus as claimed in claim 15 wherein the model 
generating means is operable to generate surface elements 
of the model from the model data including the additional 
model point and wherein the control means is operable to 
control the display to display said surface elements in 
the model image . 

17. A computer program comprising processor 
implementable instructions for carrying out a method as 
claimed in any of claims 1 to 8. 
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18. A storage medium storing processor implementable 
instructions for controlling a processor to carry out a 
method as claimed in any of claims 1 to 8 - 

5 19- An electrical signal. carrying processor 
implementable instructions for controlling a processor to 
carry out a method as claimed in any of claims 1 to 8 . 

20. A method of operating an apparatus for generating 
10 model data defining a model in a three dimensional space, 
the model data comprising co-ordinates defining model 
points and surface elements generated with reference to 
the model points; the method comprising editing an 
existing set of model data by the steps of; 
15 adding a new model point to the existing set of 

model data; 

projecting the new model point onto the model and 
identifying a selected one of the surface elements onto 
which the new model point is projected; 
2 0 identifying a subset of the model points which 

define the generation of the selected surface element; 

adding the new model point to the subset to form an 
edited subset of model points; and 

generating one or more edited surface elements from 
25 the edited subset of model points to replace the selected 
surface element. 
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21- A method as claimed in claim 20 wherein the 
projecting step comprises receiving input data defining 
a centre of projection and projecting the new model point 
onto the model in a direction of projection along a ray 
generated through the centre of projection and the new 
model point. 

22. A method as claimed in claim 21 wherein the existing 
set of model data is generated by processing image data 
representative of camera images of an object to be 
modelled. 

23. A method as claimed in claim 22 wherein the step of 
receiving input data comprises receiving an image 
selection signal for selecting one of said camera images, 
and defining the centre of projection for projecting the 
model point to be co-ordinates representative of a camera 
position from which the selected camera image was taken. 

24. A method as claimed in claim 23 including the step 
of displaying a set of camera images and receiving a 
selection signal responsive to user actuation of an input 
means to select the selected csonera image. 

25. A method as claimed in claim 22 wherein the step of 
receiving input data comprises receiving an image 



74 



263620\ 



selection signal for selecting one of said camera images, 
and receiving an image point selection signal defining 
co-ordinates of an image point in said selected camera 
image corresponding to the new model point - 

26. A method as claimed in claim 25 including the step 
of calculating in the three dimensional space co- 
ordinates of the centre of projection to correspond to 
the position of the image point in an image plane of the 
camera. 

27. A method as claimed in any of claims 21 to 26 
including the step of determining whether a plurality of 
surface elements are intersected by the ray and, if so, 
determining the selected surface to be whichever of the 
intersected surface elements is closest to the new model 
point . 

28. A method as claimed in any of claims 20 to 27 
wherein the surface elements comprise triangular facets 
and wherein each subset of the model points defining the 
generation of the selected surface element comprises 
three model points constituting apices of the triangular 
facets. 

29. Apparatus for generating model data defining a model 
in a three dimensional space, the model data comprising 
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co-ordinates defining model points and surface elements 
generated with reference to the model points, the 
apparatus being operable to edit an existing set of model 
data and comprising; 

means for adding a new model point to the existing 
set of model data; 

means for projecting the new model point onto the 
model and identifying a selected one of the surface 
elements onto which the new model point is projected; 

means for identifying a subset of the model points 
which define the generation of the selected surface 
element; 

means for adding the new model point to the subset 
to form an edited subset of model points; and 

means for generating one or more edited surface 
elements from the edited subset of model points to 
replace the selected surface element. 

30. Apparatus as claimed in claim 29 wherein the 
projecting means comprises receiving means for receiving 
input data defining a centre of projection, the 
projecting means being operable to project the new model 
point onto the model in a direction of projection along 
a ray generated through the centre of projection and the 
new model point. 

31. Apparatus as claimed in claim 3 0 wherein the 
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existing set of model data is generated by processing 
image data representative of camera images of an object 
to be modelled. 

32. Apparatus as claimed in claim 31 wherein the 
receiving means is operable to receive an image selection 
signal for selecting one of said camera images^ and to 
define the centre of projection for projecting the model 
point to be co-ordinates representative of a camera 
position from which the selected camera image was taken. 

33. Apparatus as claimed in claim 32 comprising 
interface means for displaying a set of camera images and 
receiving a selection signal responsive to user actuation 
of an input means to select the selected camera image. 

34. Apparatus as claimed in claim 31 wherein the 
receiving means is operable to receive an image selection 
signal for selecting one of said camera images, and to 
receive an image point selection signal defining co- 
ordinates of an image point in said selected camera image 
corresponding ^to the new model point. 

35. Apparatus as claimed in claim 34 including 
calculating means for calculating in the three 
dimensional space co-ordinates of the centre of 
projection to correspond to the position of the image 
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point in an image plane of the camera, 

36. Apparatus as claimed in any of claims 30 to 35 
including means for determining whether a plurality of 
5 surface elements are intersected by the ray and, if so, 
determining the selected surface to be whichever of the 
intersected surface elements is closest to the new model 
point. 

10 37. Apparatus as claimed in any of claims 30 to 36 
wherein the surface elements comprise triangular facets 
and wherein each subset of the model points defining the 
generation of the selected surface element comprises 
three model points constituting apices of the triangular 

15 facets. 

38. A computer progrcim comprising processor 
implementable instructions for carrying out a method as 
claimed in any of claims 20 to 28. 

20 

39. A storage medium storing processor implementable 
instructions for controlling a processor to carry out a 
method as claimed in any of claims 20 to 28. 

25 40. An electrical signal carrying processor 
implementable instructions for controlling a processor to 
carry out a method as claimed in any of claims 20 to 28. 
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41. A method of operating an apparatus for generating 
model data representative of a three dimensional model of 
an object from input signals representative of a set of 
camera images of the object taken from a plurality of 
5 camera positions, the method comprising; 

displaying a set of icons, each being associated 
with a respective one of the camera images of the object; 

receiving a selection signal responsive to user 
actuation of an input means whereby the selection signal 
10 identifies a selected one of the icons; 

determining a selected camera image from the set of 
ccunera images corresponding to the selected icon; 

displaying the selected image; 

determining position data representative of a 
15 selected camera position from which the selected image 
was taken; 

generating in accordance with said model a model 
image representative of a view of the model from a 
viewpoint corresponding to the position data; and 
20 displaying the model image for visual comparison 

with the selected image by the user. 

42- A method as claimed in claim 41 including the step 
of generating the icons in response to receiving a mode 
2 5 selection input. 



43. A method as claimed in any of claims 41 and 42 



v^herein the icons are generated as thumbnail images of 
the respective camera images. 

44. A method as claimed in claim 43 wherein the step of 
displaying the set of icons comprises displaying the 
icons in an array and displaying links between the icons 
such that each pair of icons corresponding to adjacent 
camera positions in a positional sequence of the camera 
positions is joined by a respective link. 

45. A method as claimed in claim 44 wherein the icons 
are displayed in a linear array. 

46. A method as claimed in any of claims 41 to 45 
wherein the selected camera image and the model image are 
displayed in respective windows and including the step of 
providing relative movement of the windows in response to 
receiving window movement input signals. 

47. A method as cladLmed in claim 46 wherein the icons 
are displayed in a further window and including the step 
of facilitating movement of the further window relative 
to the image windows in response to window movement input 
signals . 

48. A method as claimed in any of claims 41 to 47 
comprising generating the selection signal by operation 
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of a pointing means for user actuation in selecting one 
of the displayed icons . 

49. A method as claimed in claim 41 wherein displaying 
the set of icons comprises displaying a view of the model 
from a viewpoint in which the icons comprise 
representations of cameras and are shown at respective 
positions relative to the model which correspond 
substantially to the camera positions relative to the 
object. 

50- Apparatus for generating model data representative 
of a three dimensional model of an object from input 
signals representative of a set of camera images of the 
object taken from a plurality of camera positions, the 
apparatus comprising; 

display means for displaying a set of icons, each 
being associated with a respective one of the camera 
images of the object; 

means for receiving a selection signal responsive to 
user actuation of an input means whereby the selection 
signal identifies a selected one of the icons; 

means for determining a selected camera image from 
the set of camera images corresponding to the selected 
icon whereby the display means is operable to display the 
selected image; 

means for determining position data representative 
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of a selected camera position from which the selected 
image was taken; 

means for generating in accordance with said model 
a model image representative of a view of the model from 
a viewpoint corresponding to the position data; and 
control means for controlling the display means to 
display the model image for visual comparison with the 
selected image by the user- 

51. Apparatus as claimed in claim 50 further comprising 
means for generating the icons in response to receiving 
a mode selection input. 

52. Apparatus as claimed in any of claims 50 and 51 
wherein icon generating means is operable to generate the 
icons as thumbnail images of the respective camera 
images. 

53. Apparatus as claimed in claim 52 wherein the control 
means is operable to control the display means to display 
the set of icons in an array and to display links between 
the icons such that each pair of icons corresponding to 
adjacent camera positions in a positional sequence of the 
camera positions is joined by a respective link. 

54. Apparatus as claimed in claim 53 wherein the control 
means is operable to control the display means to 
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display the icons in a linear array. 

55. Apparatus as claimed in any of claims 50 to 54 
wherein control means is operable control the display 
5 means to display the selected ccimera image and the model 
image in respective windows and to provide relative 
movement of the windows in response to receiving window 
movement input signals - 

10 56. Apparatus as claimed in claim 55 wherein the control 
means is operable to control the display means to display 
the icons in a further window to facilitate movement of 
the further window relative to the camera image, window 
and model image window in response to window movement 

15 input signals. 

57. Apparatus as claimed in any of claims 50 to 56 
wherein the means for generating the selection signal 
comprises a pointing means for user actuation in 
20 selecting one of the displayed icons. 

58 . , Apparatus as claimed in claim 50 wherein the control 
means is operable to control the display means for 
displaying the set of icons by displaying a view of the 
2 5 model from a viewpoint in which the icons comprise 
representations of cameras and are shown at respective 
positions relative to the model which correspond 
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substantially to the camera positions relative to the 
object . 

59. A computer program comprising processor 
implementable instructions for carrying out a method as 
claimed in any of claims 41 to 49. 

60. A storage medium storing processor implementable 
instructions for controlling a processor to carry out a 
method as claimed in any of claims 41 to 49. 

61. An electrical signal carrying processor 
implementable instructions for controlling a processor to 
carry out a method as claimed in any of claims 41 to 49. 

62. In a method of operating an apparatus for generating 
model data representative of a model in a three 
dimensional space from image data representative of a set 
of camera images of an object, an improvement comprising; 

the apparatus performing the steps of; 
displaying a model, image based on an existing set of 
model data; 

displaying one of the camera images of the object 
for selection by a user of an additional feature to be 
represented by additional model data; 

receiving an image point selection signal responsive 
to user actuation of an input means and identifying co- 
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ordinates of an image point in the camera image defining 
the selected additional feature; 

calculating a locus in the three dimensional space 
defining positions of possible model points corresponding 
5 to the image point and consistent with the geometric 
relationship between the object and a camera position 
from which the displayed camera image was taken; 

displaying a position indicator in the model image 
at co-ordinates in the model image corresponding to one 
10 of the possible model points on the locus; 

receiving positioning signals responsive to user 
actuation of the input means and updating the co- 
ordinates of the position indicator such that movement of 
the position indicator is constrained to follow a 
15 trajectory in the model image corresponding to the locus; 

receiving a model point selecting signal responsive 
to user actuation of the input means and determining 
selected co-ordinates of the position indicator to be the 
position indicator co-ordinates at the time of receiving 
20 the model point selecting signal; and 

determining co-ordinates of the additional model 
point in thet three dimensional space corresponding to the 
selected co-ordinates of the position indicator . 

25 63. In an apparatus for generating model data 
representative of a model in a three dimensional space 
from image data representative of a set of camera images 
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of an object; 

an improvement wherein the apparatus comprises; 
an interface comprising display means operable to 
display images to a user and input means responsive to 
5 user actuation; 

control means operable to control the display means 
to display a model image based on an existing set of 
model data and to display one of the camera images of the 
object for selection by a user of an additional feature 
10 to be represented by additional model data; 

receiving means for receiving an image point 
selection signal responsive to user actuation of the 
input means and identifying co-ordinates of an image 
point in the camera image defining the selected 
15 additional feature; 

calculating means for calculating a locus in the 
three dimensional space defining positions of possible 
model points corresponding to the image point and 
consistent with the geometric relationship between the 
2 0 ob j ect and a camera position from which the displayed 
camera image was taken; 

the control means being further operable to control 
the display means to display a position indicator in the 
model image at co-ordinates in the model image 
25 corresponding to one of the possible model points on the 
locus ; 

the apparatus further comprising means for receiving 
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positioning signals responsive to user actuation of the 
input means and updating the co-ordinates of the position 
indicator such that movement of the position indicator is 
constrained to follow a trajectory in the model image 
5 corresponding to the locus; 

means for receiving a model point selecting signal 
responsive to user actuation of the input means and 
determining selected co-ordinates of the position 
indicator to be the position indicator co-ordinates at 
10 the time of receiving the model point selecting signal; 
and 

means for determining co-ordinates of the additional 
model point in the three dimensional space corresponding 
to the selected co-ordinates of the position indicator* 

15 

64. In an apparatus for generating model data 
representative of a model in a three dimensional space 
from image data representative of a set of camera images 
of an object, a method wherein; 
2 0 the apparatus performs the steps of; 

displaying a model image based on an existing set of 
. model? datk ^c ; - - 

displaying one of the camera images of the object 
for selection by a user of an additional feature to be 
25 represented by additional model data; 

receiving an image point selection signal responsive 
to user actuation of an input means and identifying co- 
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ordinates of an image point in the camera image defining 
the selected additional feature; 

calculating a locus in the three dimensional space 
defining positions of possible model points corresponding 
to the image point and consistent with the geometric 
relationship between the object and a camera position 
from which the displayed camera image was taken; 

displaying a position indicator in the model image 
at co-ordinates in the model image corresponding to one 
of the possible model points on the locus; 

receiving positioning signals responsive to user 
actuation of the input means and updating the co- 
ordinates of the position indicator such that movement of 
the position indicator is constrained to follow a 
trajectory in the model image corresponding to the locus; 

receiving a model point selecting signal responsive 
to user actuation of the input means and determining 
selected co-ordinates of the position indicator to be the 
position indicator co-ordinates at the time of receiving 
the model point selecting signal; and 

determining co-ordinates of the additional model 
point in the three dimensional space corresponding to the 
selected co-ordinates of the position indicator. 

65. In a method of operating an apparatus for generating 
model data defining a model in a three dimensional space, 
the model data comprising co-ordinates defining model 
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points and surface elements generated with reference to 
the model points; an improvement wherein the method 
comprises editing an existing set of model data by the 
steps of; 

adding a new model point to the existing set of 
model data; 

projecting the new model point onto the model and 
identifying a selected one of the surface elements onto 
which the new model point is projected; 

identifying a subset of the model points which 
define the generation of the selected surface element; 

adding the new model point to the subset to form an 
edited subset of model points; and 

generating one or more edited surface elements from 
the edited subset of model points to replace the selected 
surface element. 

66. In an apparatus for generating model data defining 
a model in a three dimensional space, the model data 
comprising co-ordinates defining model points and surface 
elementsjgenerated with reference to the model points , an 
imprdvemefit^^^ w^^ apparatus is operable to edit an 

existing set of model data and comprises; 

means for adding a new model point to the existing 
set of model data; 

means for projecting the new model point onto the 
model and identifying a selected one of the surface 
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elements onto which the new model point is projected; 

means for identifying a subset of the model points 
which define the generation of the selected surface 
element; 

means for adding the new model point to the subset 
to form an edited subset of model points; and 

means for generating one or more edited surface 
elements from the edited subset of model points to 
replace the selected surface element. 

67. In an apparatus for generating model data defining 
a model in a three dimensional space, the model data 
comprising co-ordinates defining model points and surf ace 
elements generated with reference to the model points; a 
method comprising editing an existing set of model data 
by the steps of; 

adding a new model point to the existing set of 
model data; 

projecting the new model point onto the model and 
identifying a selected one of the surface elements onto 
which the new model point is projected; 

identifying a subset of the model points which 
define the generation of the selected surface element; 

adding the new model point to the subset to form an 
edited subset of model points; and 

generating one or more edited surface elements from 
the edited subset of model points to replace the selected 
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surface element . 

69. In a method of operating an apparatus for generating 
model data representative of a three dimensional model of 
an object from input signals representative of a set of 
camera images of the object taken from a plurality of 
ccimera positions^ an improvement wherein the method 
comprises; 

displaying a set of icons, each being associated 
with a respective one of the camera images of the object; 

receiving a selection signal responsive to user 
actuation of an input means whereby the selection signal 
identifies a selected one of the icons; 

determining a selected camera image from the set of 
camera images corresponding to the selected icon; 

displaying the sielected image; 

determining position data representative of a 
selected c£anera position from which the selected image 
was taken; 

generating in accordance with said model a model 
image representative of a view of the model from a 
viewpoint corresponding to the position data; and 

displaying the model image for visual- comparison 
with the selected image by the user. 

70. In an apparatus for generating model data 
representative of a three dimensional model of an object 
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from input signals representative of a set of camera 
images of the object taken from a plurality of camera 
positions^ an improvement wherein the apparatus 
comprises; 

display means for displaying a set of icons, each 
being associated with a respective one of the camera 
images of the object; 

means for receiving a selection signal responsive to 
user actuation of an input means whereby the selection 
signal identifies a selected one of the icons; 

means for deteirmining a selected camera image from 
the set of camera images corresponding to the selected 
icon whereby the display means is operable to display the 
selected image; 

means for determining position data representative 
of a selected ceunera position from which the selected 
image was taken; 

means for generating in accordance with said model 
a model image representative of a view of the model from 
a viewpoint corresponding to the position data; and 
control means for controlling the display means to 
. display the model image for visual comparison with the 
selected image by the user. 

71. In an apparatus for generating model data 
representative of a three dimensional model of an object 
from input signals representative of a set of camera 
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images of the object taken from a plurality of camera 
positions, a method comprising; 

displaying a set of icons, each being associated 
with a respective one of the camera images of the object; 

receiving a selection signal responsive to user 
actuation of an input means whereby the selection signal 
identifies a selected one of the icons; 

determining a selected camera image from the set of 
camera images corresponding to the selected icon; 

displaying the selected image; 

determining position data representative of a 
selected camera position from which the selected image 
was taken; 

generating in accordance with said model a model 
image representative of a view of the model from a 
viewpoint corresponding to the position data; and 

displaying the model image for visual comparison 
with the selected image by the user. 
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ABSTRACT 

METHOD AND APPARATUS FOR GENERATING 
MODEL DATA FROM CAMERA IMAGES 
A three-dimensional model of an object is generated 
from image data representing a set of camera images of 
the object- Refinement of the model is carried out on 
the basis of a comparison of a displayed model image and 
a displayed camera image/ enabling a user to select co- 
ordinates in the camera image defining the selection of 
an additional feature which is to be added to the model. 
The model calculates a locus in the three-dimensional 
space defining positions of possible model points 
corresponding to the selected image point and consistent 
with the geometric relationship between the object and a 
ccimera position from which the displayed ccimera image was 
taken. A position indicator is displayed in the model 
image at co-ordinates on the locus and the position 
indicator is moveable by the user in a manner which is 
constrained to follow a trajectory in the model image 
corresponding to the locus. The user may then select a 
required position of the new model point on the locus and 
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the model data is updated to include the new model point 
corresponding to such selection. The surface of the 
model may be edited to incorporate a new model point by 
identifying a facet by projection onto the model and 
automatically re-facetting the model. Evaluation of the 
model image requires presenting corresponding camera and 
model images, the model viewpoint being determined 
automatically by user selection of camera frames or 
camera position - 



(Figures 5 and 7 refer) 
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Fig 2 

MODEL WINDOW SHOWS FEATURE MISSING FROM 
MODEL IMAGE WHEN COMPARED WITH CAMERA IMAGE 
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Fig 3 IN CAMERA IMAGE WINDOW ,USER SELECTS NEW 

IMAGE POINT FOR ADDITION TO MODEL 
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Fig 4 

MODEL CALCULATES LOCUS OF NEW MODEL POINT 
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Fig 5 




IN MODEL WINDOW.MODEL IMAGE IS DISPLAYED FROM A 
DIFFERENT VIEWING ANGLE AND USER MOVES NEW MODEL 
POINT TO REQUIRED POSITION .CONSTRAINED BY 
CALCULATED LOCUS 




Fig 7 

IN MODEL WINDOW, MODEL IMAGE NOW SHOWS NEW 
MODEL POINT AND FACETS 
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Fig 8 



MODEL IMAGE WITH ADDED MODEL DATA , VIEWED FROM 
^ SAME DIRECTION AS CAMERA IMAGE 
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USER VIEWS IMAGES 
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USER SELECTS AN IMAGE 

POINT IN THE CAMERA 
IMAGE USING A POINTING 
MEANS 
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IMAGE AND POSITION 
INDICATOR 
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USER ACTUATES 
POINTING MEANS TO 
INTERACTIVELY MOVE THE 
POSITION INDICATOR 
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Fig 11 
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DISPLAY OF LINE INDICATING LOCUS/ 
TRAJECTORY 
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Fig 12 

ADDING A NEW MODEL POINT 
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Fig 13A 



CAMERA SELECTION WINDOW WITH CAMERA ICONS 
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Fig 13B 

CAMERA SELECTION WINDOW WITH CAMERA IMAGES 
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Fig 14 



CALCULATE FACET INTERSECTED BY RAY THROUGH 
CAMERA POSITION AND ADDED MODEL POINT 
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Fig 15 

SUBDIVIDE FACET TO INCLUDE ADDED MODEL POINT 
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Fig 16 

DISPLAY NEW MODEL INCLUDING ADDED POINT AND NEW FACETS 
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